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ABSTRACT 


This  thesis  quantitatively  analyzes  the  factors  that  affect  weapon 
system  cost  growth  after  Milestone  II.  The  data  from  nine  weapon  systems 
was  reconstructed  by  the  Army  and  Navy  from  Selected  Acquisition  Renorts 
(SARs)  with  the  cost  variances  reclassified  into  a  new  categorization 
system  to  more  readily  determine  the  causes  of  cost  growth.  Each  cost 
v,ariance  was  classified  as  to  whether  it  was  attributable  to  a  mistake  in 
the  cost  estimating  process  or  a  post-Milestone  II  decision,  with  further 
clarsif icaticn  into  subcategories  for  a  more  detailed  analysis.  The  cost 
variances  were  divided  by  the  Milestone  II  Decision  Estimate  (DE)  to  form 
a  cost  growth  ratio  ^CGR) .  The  findings  reveal  that  the  Department  of 
Defense  has  about  10.8%  cost  growth  in  the  procurement  process.  Cost 
grov/th  due  to  decisions  outweigh  mistakes  by  a  factor  of  2.3:1.  A 
majority  of  the  mistake  cost  grov.'th  is  due  to  errors  in  the  estimation  of 
production  costs.  A  majority  of  the  decision  cost  growth  is  due  to 
schedule  slippage.  Low  cost  systems  have  2.4  times  as  much  mistake  cost 
growth  as  high  cost  systems.  Newer  missile  systems  have  significantly 
les.s  mistake  cost  growth  when  compared  to  other  systems.  Lastly,  the  Army 
and  Navy  have  approximately  equal  cost  growth  on  their  newer  systems. 
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EXECUTIVE  SUMMARY 


A  common  perception  among  the  public,  in  Congress,  and  even  within  the 
defense  establishment  is  that  weapon  system  cost  growth  results  from  poor 
managerial  practices  combined  with  an  inability  to  accurately  estimate 
weapons  system  costs.  This  oversimplification  of  a  very  complex  problem 
is  at  the  forefront  of  discussion  now  that  reduced  budgets,  military 
downsizing,  and  decreased  procurement  are  realities. 

Many  cost  gro^^th  studies  occurred  in  the  early  1930s  in  response  to 
the  military  buildup  which  began  in  1980.  Very  little  research  has  been 
done  since  then  on  the  new  high  technology  systems.  This  lack  of  adequate 
research  combined  with  tighter  military  budgets  shows  a  need  for  new 
research  on  the  factors  affecting  cost  growth.  The  objective  of  this 
research  is  not  to  predict  or  model  particular  systems,  bat  to  determine 
those  factors  that  affect  the  cost  growth  of  the  Department  of  Defense 
(DoD)  as  a  whole. 

In  order  to  more  readily  identify  areas  of  cost  growth,  it  is 
necessary  to  classify  the  cost  variances  into  a  more  useable  and  more 
detailed  categorization  system.  The  new  categorization  system 
differentiates  between  decisions  made  that  would  knowingly  increase  weapon 
sy.stem  cost  and  mistakes  that  misestimated  aspects  in  the  process  which, 
if  estimated  correctly,  would  not  have  resulted  in  cost  growth.  The  new 
categories  also  partition  these  two  broad  categories  into  six 
subcategorie""  each  to  give  insight  into  the  most  likely  areas  for  cost 
growth  to  occur. 

The  data  was  analyzed  to  determine  if  factors  such  as  system  cost, 
service,  and  type  of  system  affect  cost  growth.  This  was  done  to  identify 
specific  areas  in  the  cost  estimation  and  procurement  processes  that  need 
attention  and  to  determine  the  probable  causes  of  there  differences. 
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The  findings  reveal  that  the  Department  of  Defense  has  experienced 
approximately  10.8%  cost  growth  in  the  procux-ement  process  after  Milestone 
II,  which  is  lower  than  might  be  expected.  Studies  done  during  the  early 
lS80s  report  that  the  average  total  cost  growth  of  weapon  systems  was 
between  20%  and  30%.  However,  the  time  of  those  studies  is  relevant  to 
the  difference  in  findings.  Three  major  changes  have  occurred  in  the  last 
decade  to  lower  weapon  system  cost  growth.  First,  estimators  have 
developed  a  large  computerized  cost  data  base  to  more  accurately  predict 
costs.  Second,  high  technology  systems,  that  were  brand  new  in  1980,  have 
been  developed  and  produced  so  tliat  they  are  now  much  easier  to  estimate. 
Lastly,  the  early  studies  resulted  in  a  determined  effort  by  the 
Department  of  Defense  to  control  cost  growtn. 

Mistakes  made  in  the  estimation  of  system  costs  make  up  30.6%  of  the 
total  cost  growth  of  a  system  while  decisions  make  up  C9.4%  of  the  total 
cost  growth.  Therefore  decisions  outweigh  mistakes  by  a  margin  of  2.3:1. 

A  majority  of  the  mistake  cost  growth  is  attributable  to  mistakes  In 
estimating  the  cost  of  production.  However,  mistake  cost  growth  averages 
only  3.3%  and  is  not  nearly  as  significant  as  controlling  the  cost  growth 
due  CO  decisions. 

A  majority  of  the  decision  cost  growth  is  attributable  to  schedule 
slippage.  Some  of  this  schedule  slippage  can  be  attributed  to  decisions 
to  change  the  design  and  performance  requirements  of  the  system,  while  the 
remaining  amount  is  unexplained.  Further  study  concentrating  on  the 
causes  of  schedule  slipp^^ge  needs  to  be  done. 

The  low  cost  systems  have  a  higher  mistake  cost  growth  when  compared 
to  high  cost  system,  by  a  margin  of  2.4:1.  Two  possible  explanations 
exist  for  this  effect.  Fi?:st,  estimators  do  not  take  as  much  care  in 
estimating  the  costs  of  low  cost  systems  since  they  are  not  as  visible  on 
an  oversight  level.  The  second  reason  concerns  the  nature  of  high  cost 
and  low  cost  systems.  A  majority  of  high  coat  systems  are  large  platforms 
which  have  a  majority  of  their  production  cost  resulting  from  large 
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components  or  similar  components  that  have  been  used  oef ore .  This  makes 
the  estimation  of  the  production  costs  of  these  large  components  fairly 
accurate.  Therefore,  the  mistakes  in  estimating  the  smaller  and 
relatively  cheaper  items  would  be  masked  in  high  cost  systems  by  these 
high  cost  items.  Therefore,  the  difference  between  the  low  cost  and  high 
cost  system  mistake  cost  growth  is  probably  not  as  significant  as  it 
appears  to  be . 

The  significant  differ  nee  between  the  cost  growth  of  new  missile 
systems  and  other  weapon  -.ems  is  that  the  mistake  cost  growth  for  new 
missile  systems  is  signi  antly  less  than  that  of  other  systems, 
specifically  in  estimating  the  cest  of  production.  This  is  due  to  the 
evolutionary  nature  of  missile  systems.  The  guidance,  propulsion,  and 
warhead  systems  of  newer  missiles  is  generally  a  modernization  or 
modification  of  a  previous  system.  This  would  tend  to  increase  the 
accuracy  of  missile  system  estimates  over  other  weapon  systems .  Previous 
studies  show  that  missile  systems  have  a  much  higher  cost  growth  than  do 
other  systems.  These  studies  included  much  older  missile  systems.  The 
evolutionary  process  discussed  above  had  not  yet  developeo  and  therefore 
a  higher  cost  growth  could  have  resulted. 

The  Army  and  Navy  have  effectively  etpial  cost  growths.  A  majority  of 
early  1980s  studies  reported  that  the  Army  had  a  significantly  higher  cost 
growth  than  the  Navy.  This  was  attributed  to  the  modernization  of  the 
Army.  This  study  concentrated  on  more  recent  weapons  systems  and  (^id  not 
find  this  to  be  the  case.  The  most  obvious  explanation  for  this  effect  is 
that  the  Army  has  completed  its  modernization  and  is  now  proficient  at 
controlling  the  costs  of  its  weapon  systems. 

These  results  are  promising  in  that  it  appears  that  over  the  last 
decade,  the  DoD  has  gained  control  over  cost  growth.  It  is  still 
necessary  to  determine  and  correct  the  cause  of  schedule  slippage,  but,  on 
the  whole,  the  cost  growth  problem  is  being  solved. 
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I .  INTRODUCTION 


A  conimon  perception  among  the  public,  in  Congress,  and  even  within 
the  defense  establishment  is  that  weapons  system  cost  growth  results 
from  poor  managerial  practices  combined  with  an  inability  to  accurately 
estimate  weapons  system  costs.  This  oversimplification  of  a  very 
complex  problem  is  at  the  forefx'ont  of  discussion  now  that  reduced 
budgets,  military  doTOSizing,  and  decreased  procurement  are  realities. 

Many  studies  have  shown  that  most  of  a  weapons  system' s  procurement 
cost  growth  occurs  after  it  enters  Engineering  and  Manufacturing 
Development  (EMD)  but  prior  to  Full  Scale  Production  [Ref.  1] .  This 
cost  growth  v.'ill  bt  magnified  if  the  United  States  attempts  to  maintain 
its  technological  edge,  in  the  reduced  budget  environment  by  developing 
technology  and  then  "shelving  it"  until  neeued.  This  will  result  in 
increased  scrutiny  by  the  Congress  on  cost  growth  and  its  control. 

Most  of  the  research  on  cost  grov/th  occurred  prior  to  the  1980' s 
military  buildup.  Some  research  focused  on  single  system  cost  gro'./th 
(e.g.,  just  the  F-14  [Ref.  2]).  This  was  done  to  identify  and  change 
managerial  and  estimating  problems  within  a  particular  weapons  system 
already  identified  as  having  high  cost  growth.  The  remaining  research 
used  the  Selected  Acquisition  Report  (SAR)  categories  for  a  basis  [Ref 
3] .  In  the  SAR,  each  cost  variance  is  categorized  as  an  estimating, 
economic,  quantity',  schedule,  support  or  other  change  from  the  Milestone 
II  baseline  estimate.  Unfortunately,  these  categories  were  not 
uniformly  used  across  different  systems  to  classify  cost  variances, 
making  any  cross-system  analysis  using  these  categories  of  limited 
validity.  Furthermore,  the  criteria  for  classifying  a  cost  variance 
into  one  of  these  SAR  categories  changed  in  the  mid-1980's  and  as  a 
result,  comparisons  across  this  time  are  not  valid.  Therefore,  none  of 
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the  relatively  new  high  technology  systems  can  be  analyzed  using  a  time 
series  approach. 

As  a  result  of  this  lac.t  of  adequate  data  on  current  technology 
combined  with  tighter  military  budgets,  new  research  on  the  factors 
affecting  cost  growth  is  required.  It  will  be  necessary  to  form  new 
categorization  criteria  for  cost  variances  in  order  to  break  the  cost 
variances  into  a  more  useful  structure.  The  objective  of  this  research 
is  not  to  predict  or  model  particular  systems,  but  to  determine  those 
factors  that  affect  the  cost  growth  of  the  Department  of  Defense  as  a 
whole.  The  quantitative  analysis  of  this  data  is  designed  to  yield 
descriptive  information  on  those  areas  in  the  procurement  process  where 
cost  growth  is  occurring.  This  information  can  be  used  by  Program 
Managers  and  cost  estimators  to  evaluate  their  cost  methodology  and 
refine  their  cost  estimating  techniques.  This  should  ultimately  yield 
lower  procurement  cost  growth. 
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II .  BACKGROUND 


A.  REVIEW  OF  THE  ACQUISITION  PROCESS 

The  purpose  of  this  chapter  is  to  provide  a  brief  review  of  Che 
weapons  system  acg^iisition  process  prior  to  a  discussion  of  research 
methodology  used  and  why  data  from  specific  points  in  the  acquisition 
process  was  used. 

The  acquisition  process  i.s  broken  down  into  five  phases  with  the 
transition  between  phases  being  marked  by  a  milestone.  Decision 
authorities  determine  at  each  milestone  whether  the  results  of  the 
previous  phase  warrant  continuation  into  the  next  acquisition  phase. 
Department  of  Defense  Instruction  5000.2  (current  version  dated  February 
1991)  .iGscribes  the  process  and  discusses  the  objectives  of  each  phase 
and  milestone  in  the  acq-aisition  process.  Systems  within  the 
acquisition  process  do  I’.oL  necessarily  follow  this  exact  process 
However,  each  system  must  ultimately  gain  approval  at  each  milestone  in 

this  process. 

1.  Mission  Need 

Initially,  a  determination  of  mission  need  is  decided  upon,  by 
various  sources,  based  ui:ion  continuing  nsses,sments  of  tlie  military 

requirements  and  capabilities. 

2.  Milestone  0,  Concept  Studies  Approval 

Decision  authorities  determine  if  a  study  of  alternatives  is 
warranted  and  determine  the  alternatives  to  be  evaluated.  If  approval 

is  granted,  the  system  proceeds  into  Phase  0. 

3.  Phase  0,  Concept  Exploration  and  Definition 

Various  alternatives  are  evaluated.  The  most  promising  system..^ 
satisfying  the  mission  requirements  are  evaluated  to  determine  high  riel' 
areas  and  initial  objectives  for  program  cost,  schedule,  and 
performance , 
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4.  Milestone  I,  Concept  Demonstration  Approval 


Decision  authorities  determine  if  a  new  acquisition  program  is 
needed  based  on  the  results  of  Phase  0.  A  baseline  for  program  cost, 
schedule,  and  perf c-mance  is  established.  Approval  at  Milestone  I 

authorizes  the  program  to  move  into  Phase  I. 

5.  Phase  I,  Demonstration  and  Validation 

Critical  design  and  performance  characteristics  are  thoroughly 

evaluated.  Schedule,  cost,  and  performance  objectives  are  refined. 

6.  Milestone  XI,  Development  Approval 

If  the  results  of  Phase  I  warrant  continuation,  a  Developmental 
Baseline  for  cost,  schedule  and  performance  for  the  program  is  approved. 
This  includes  the  Developmental  Estimate  (DE)  v.'hich  defines  the 
objective  total  system  cost.  This  estimate  i.s  the  refined  total  cost 
from  Milestone  I.  Milestone  II  approval  authorizes  program  continuation 
into  Phase  II. 

7.  Phase  II,  Engineering  and  Manufacturing  Development 

The  .system  enters  full  scale  engineering  development.  A  cost 
effective,  producible  design  is  developed  to  validate  the  production 
process.  Tests  are  conducted  to  verify  that  system  performance 

satisfies  performance  requirements. 

8.  Milestcna  III,  Production  Approval 

A  production  baseline  for  cost,  schedule  and  performance  using 
the  revised  data  from  Phase  II  is  eotabl islied .  The  system  now  proceeds 
into  Phase  III. 

9.  Phaaa  III,  Production  and  Doployioent/Phaae  IV,  Oporationc  and 
Support 

All  efficient  production  capaliility  along  v/ith  its  associated 
.support  ba.se  i.s  c.stabliahed  to  achieve  operational  oapiability  oati.sfying 
the  mission  need.  Once  initial  systems  are  fielded,  system  monitoring, 
support  and  problem  correction  continue  throughout  system  lifetime. 
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10.  Milestone  XV,  Major  Modification  Approval 

Decision  authorities  determine  if  a  major  modification  or 
change  to  the  system  is  warranted  and  if  so,  how  to  implement  this 
change  in  the  most  effective  manner. 

B.  SELECTED  ACQUISITION  REPORT  (SAR) 

The  data  used  is  derived  from  the  Selected  Acquisition  Report  (SAR) . 
Comprehensive  instructions  for  the  SAR  are  contained  in  Department  of 
Defense  Manual  5000. 2M.  The  SAR  is  a  comprehensive  report  designed  to 
provide  Defense  Department  officials  with  cost  and  management 
information  on  each  major  weapons  system.  The  SARs  are  submitted  to  the 
Congress  to  allow  the  Armed  Services  Committees  to  monitor  the 
Departme.nt '  s  progress  in  meeting  its  procurement  plans,  and  to  provide 
an  early  warning  of  emerging  cost  problems. 

The  SARs  are  compilations  of  status  reports  from  the  Program 
Managers  responsible  for  major  defense  acquisition  programs.  Th  ;y 
provide  each  Program  Manager's  latest;  est-.imates  of  progress  in  achieving 
key  goals  with  respect  to  performance,  schedule  and  cost.  The  most 
recent  estitAate.s  are  recorded  in  then  year  dollars  and  in  constant  base 
year  drJ.lars,  with  the  base  year  generally  being  the  Milestone  II  year. 

S/iRs  are  required  by  Department  of  Defense  Manual  5000. 2M  to  be 
submitted  for  programs  that  have  been  designated  by  the  Secretary  of 
Defense  as  major  systems  or  are  estimated  to  cost  more  than  0200  million 
for  research,  development,  test,  and  evaluation  or  more  than  $1  billion 
for  procurement.  Highly  classified  programs  arc  excluded  from  this 
requirement.  .‘iARs  are  prepircd  on  an  exception  basis  for  the  first, 
second,  and  third  quarter  of  each  year,  with  a  comiJi ehens ive  report  for 
tlie  fourth  quarter  ending  Decetitber  31.  The  coat  data  iti  tlie  Dcceiidjor 
,'jAlis  are  ('xpocted  to  correspond  to  data  in  the  President ' .s  annu.al  budget 
submitted  to  the  Congress  in  January. 


1.  Categories 

The  SAR  reports  the  Program  Manager's  most  recent  best  cost 
estimate.  The  initial  estimate  listed  in  the  SAR  is  the  baseline 
estimate  or  developmental  estimate  (DE) .  The  DE  is  generally  made  at 
Milestone  II  in  the  acquisition  prcce.os.  The  SAR  records  all  changes  in 
terms  of  deviation  from  the  previous  cost  estimate.  These  changes  are 
defined  a.s  cost  variances.  The  sum  of  all  the  cost  variances  and  the  DE 
is  defined  as  the  Current  Estimate  (CE) . 

Each  cost  variance  listed  in  the  SAR  is  categorized  ;\ccordLng  to 
the  cause  of  change.  These  categories  are  explicitly  defined  in 

Departm.ent  of  Defense  Manual  5000. 2M. 
a.  Quantity  Change 

A  cost  variance  is  classified  as  a  quantity  change  if  it  xs 

due  to  a  change  in  the  number  of  units  of  an  end  item  of  equipment. 

Jb.  Schedule  Change 
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due  to  a  change  in  procurement  or  delivery  schedule,  completion  date,  or 

intermediate  milestone  for  development  or  production. 

c.  Eaglneurlag  Change 

A  cost  variance  is  classified  as  an  engineering  change  if  it 
is  due  to  an  alteration  in  the  physical  or  function  characteristics  of  a 
system  or  item  delivered,  to  be  delivered,  or  under  development,  after 
establ i.shnient  of  such  characteristics. 

d.  Economic  Change 

A  coot  variance  i.s  classified  as  an  economic  change  if  it  is 

due  to  price  level  changes  in  the  economy. 
a.  Batimating  Change 

A  coot  variance  i.s  classified  as  an  estimating  change  it  it 
i.s  tor  correction  of  an  error,  refinement  of  a  t)rior  Current,  Estintato, 
or  a  change  in  program  or  coot  e.suimatitig  assumptions  and  techniques. 
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f.  Support  Change 

A  cost  variance  is  classified  as  a  support  change  if  it  ;'.s 
due  to  a  change  in  cost,  regardless  of  reason,  associated  with  any  work 
breakdown  structure  element  not  included  in  the  flyaway,  rollaway,  or 

sailav;ay  cost. 

g.  Other  Change 

A  cost  variance  is  classified  in  the  other  change  category 
if  it  cannot  be  classified  into  one  of  the  above  categories. 
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III.  ISSOBS  PBRTXIHZNa  TO  THX  DATA 


A .  PROCORBMJHST  COSTS 

Weapon  system  costs  are  broken  down  into  tv/o  major  cost  categories, 
Research,  Development,  Testing  and  Evaluation  (RDT&E)  costs  and 
procurement  costs.  This  study  will  focus  on  only  the  procurement  cost 
growth  of  major  weapons  systems.  RDXaE  cost  growth  is  difficult  to 
quantify  because  of  constantly  changing  requirements  and  modifications 
in  the  initial  design  of  the  system.  These  changes,  which  occur  prior 
to  milestone  II,  could  greatly  affect  the  RDTfcE  cost  growth. 

Also,  the  major  focus  of  this  research  is  to  identify  managerial  and 
coat  estimation  problems  within  the  Department  of  Defense  after  a  system 
enters  EMD.  Analysis  of  RDT&E  cost  growth  would  be  more  beneficial  in 
defining  areas  that  need  attention  within  the  initial  stages  of  concept 
e.-ploration  and  definition. 

a.  fviXff  CATSGORZCAL  DSrZNITIOMS 

The  criteria  for  classifying  cost  variances  into  the  SAR  categories 
has  changed  over  the  years.  This  makes  any  cross  system  analysis  of 
lirn: '•cd  validity.  Also,  the  SAR  categories  do  not  allow  for  easy 
identification  of  the  root  causes  behind  cost  growth  of  a  particular 
weapon  system  [Ref.  4)  ,  For  e.x.ample.  a  coat  variance  ir.  the  support 
category  does  not  show  whethe.v.'  the  cost  variance  wau  due  to  a  derision 
to  change  the  support  i equi.rementa  or  if  a  niiscake  was  mads  in  initial 
estimate  of  the  support  costs.  Therefore,  the  cost  variances  need  to  be 
reclassified  into  a  new  categorization  format  before  any  a.nalysis  of  the 
reasons  for  cost  g;owth  can  be  done. 

The  new  categorization  system  used  ia  under  developm'*  .!.  by  the 
Department  of  Defense  (DoD)  ,  The  new  categories  used  to  clas -ify  cost 
varianceo  must  differentiate  between  decisions  tiiat  arc  made  that  would 
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knowingly  increase  weapon  system  cost  (i.e.,  increased  range  or 
performance)  and  mistakes  that  underestimated  or  overestimated  aspects 
in  the  process  which,  if  estimated  correctly,  would  not  have  resulted 
in  cost  growth.  The  new  categories  must  also  partition  these  broad 
categories  into  smaller  subcategories  to  give  insight  into  the  most 
likely  areas  for  cost  growth  to  occur. 

The  new  categories  consist  of  two  major  categories  defined  as  to 
whether  the  cost  variance  was  due  to  a  decision  or  a  mistake.  The  cost 
variance  is  further  classified  into  one  of  six  subcategories  according 

to  the  specific  area  that  is  related. 

1.  DAclalon  Varianc* 

A  decision  variance  is  a  cost  variance  due  to  a  decision 
external  to  a  program- s  defined  Milestone  II  baseliiie.  Examples  include 
changes  in  system  capability  and  acquisition  strategy  changes  (such  as 
dual  source  procurement,  multiyear  procurement,  etc.)  not  dictated  by 
fact  of  life  conditions.  Decision  variances  are  subcategorizea  uy  the 

type  of  change.  These  subcategories  are: 

a .  Deev 

A  cost  variance  is  classified  in  the  Derv  category  if  it 
results  from  a  decision  that  changes  the  system  requirements  or  results 

in  a  new  variant  of  the  system. 

b.  UBmmi 

A  cost  variance  i.s  classified  in  the  Psmmi  category  if  it 
results  from  a  decision  that  causes  ciiangos  to  the  procuiement  schedule, 
shifts  in  the  multiyear  procurement,  rate  or  in  different  t\xanagement 

initiatives . 

c.  Dilm 

A  cost  variance  is  classified  in  the  Oils  category  if  it 
results  from  a  decision  to  cliange  the  Integrated  Logistical  Support 
(ILS)  factors  or  changes  in  spares  or  support  requirements. 
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d.  Dmpt 


A  cost  %-ariance  is  classified  in  the  Depf  category  if  it 
results  from  changes  to  the  external  program  factors  (foreign  military 

sales  (FMS) ,  labor  strikes,  etc) . 

•.  Dmacl 

A  cost  variance  is  classified  in  the  Descl  category  if  it 
results  from  a  decision  to  change  the  escalation  (inflationary) 

requirements . 

f.  Dothar 

This  category  is  designed  to  account  for  minor  or  unforeseen 
decision  variances  that  cannot  be  categorized  into  one  of  the  above 

categories . 

2 .  Mistake  Variance 

A  mistake  variance  is  a  cost  variance  not  attributable  to  post- 
Milestone  II  decisions.  Examples  include  method  errors,  omi^isions, 
schedule  slips  attributable  to  technical  problems,  weight  growth,  and 
inadequately  scoped  engineering  and  software  development  efforts.  An 
important  point  is  that  this  category  is  considerably  broader  than 
estimating  error:  "mistakes"  include  many  factors  that  are  mani.festly 
not  considered  to  be  related  to  "cost  estimating"  (e.g.,  weight  growth) . 
This  category  is  intended  to  capture  all  the  variance  that  the 
acquisition  system  as  a  whole  should  have  anticipated,  not  just  the  cost 

estimator.  Mistake  variances  are  subcategorized  by  the  type  of  change. 

a.  Heap 

A  cost  variance  i.a  classified  in  tlie  Mcep  category  if  it  is 

due  to  a  mistake  in  estimating  the  production  costs  of  the  system. 

b.  Meade 

A  cost  variance  is  classified  in  the  Mcede  category  if  it  is 
due  to  a  mistake  in  estimating  the  developmental  and  engineering  costs 
of  the  system. 
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C.  MilB 


A  cost  variance  is  classified  in  the  Mils  category  if  it  is 
due  to  a  mistake  in  estimating  ILS  factors,  spares  or  support 

requirements . 

d .  Maamf 

A  cost  variance  is  classified  in  the  Mssmf  category  if  it  is 
due  to  schedule  slips  and  or  changes  in  the  management  factors  that  are 

not  attributable  to  a  decision. 

e.  Mead 

A  cost  variance  is  classified  in  the  Mescl  category  if  it  is 

caused  by  a  mistake  in  estimating  the  escalation  requirements. 

f.  Mother 

This  category  is  designed  to  account  for  minor  or  unforeseen 
mistake  variances  that  cannot  be  categorized  into  one  of  the  above 
categories . 

This  new  classification  system  will  allow  for  a  more  detailed 
evaluation  of  the  factors  affecting  cost  growth.  It  will  indicate  the 
specific  areas  in  which  cost  grov/th  is  occurring  and  where  measures  to 
control  cost  growth  should  be  focused. 

C.  USING  SAR  DATA 

Each  service  was  tasked  by  the  DoD  to  reconstruct  the  data  into  the 

new  categories  using  historical  SAR  data. 

1 .  Advantages  of  SAR  Data 

A  majority  of  major  weapons  systems  are  required  to  have  a 
SAR  filed  on  them.  This  allows  for  a  comprehensive  analysis  of  cost 
growth  within  the  Department  of  Defense.  Most  SARs  are  unclassified 
and  are  publicly  available.  This  allows  wide  dissemination  of  data 
and  results.  This  allows  objective  research  outside  the  government 
establishment  to  review  the  progress  on  the  control  of  cost  growth. 
Failure  to  analyze  classified  systems  should  not  significantly  impact 
the  results  of  a  cost  growth  study  since  there  are  a  limited  number  of 
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these  systems.  Furthermore,  subsequent  analyses  by  the  Office  of  the 
Secretary  of  Defense  of  all  weapons  systems  when  they  become  available, 
including  classified  systems,  will  alleviate  this  concern. 

Each  cost  variance  must  be  reported  on  the  SAR  along  with  the 
Cause  of  the  variance.  This  allows  for  a  full  accounting  by  the 
services  of  these  variances.  A  minimal  amount  of  cross  referencing  will 
be  required  by  the  services  to  obtain  the  root  cause  of  the  cost 
Variance  which  is  needed  to  accurately'  analyze  cos'-  growth  [Ref  4]  . 

These  advantages  make  the  use  of  SAR  data  a  necessity  once  it  is 
appropriately  modified  by  the  services  into  a  more  useful  categorization 

of  the  variances . 

2.  Disadvantages  o£  SAR  Data 

Since  the  old  SAR  categories  were  not  consistently  applied, 
obtaining  the  data  in  a  consistent  categorization  will  require  the 
cooperation  of  the  services  to  reclassify  the  data  into  the  new 
categories.  bach  service  will  also  need  to  reconstruct  the  actual  cause 
of  each  variance.  Service  cooperation  is  available,  although  it  will 
take  two  years  to  obtain  all  weapons  systems  covered  by  SAR  reporting 
requirements . 

The  categorization  of  the  cost  variances  is  a  judgment  call. 

The  decisions  by  individual  services  may  generate  associated  errors  that 
cannot  be  objectively  described. 

These  disadvantages,  while  important,  do  not  constitute  a  large 
problem.  The  errors  imparted  into  the  data  should  not  be  of  a 
sufficiently  large  magnitude  to  nullify  the  findings.  The  additional 
time  to  categorize  the  cost  variances  should  be  offset  by  the  time  saved 
in  using  tlie  SARs  as  described  above. 

D.  MILESTONE  II  BASELINE 

This  baseline  cost  estimate  is  considered  valid  in  the  Department  of 
Defense  since  this  is  where  the  Developmental  Estimate  (DE)  originates. 
Prior  to  this  point,  numerous  alternatives  combined  with  unproven 
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concepts  would  tend  to  introduce  gross  margins  of  error  into  any 
estimate , 

A  major  problem  when  analyzing  weapons  system  cost  growth  is  that 
the  DE  baseline  can  change.  The  baseline  is  allowed  to  change  if  major 
modifications  or  changes  to  the  weapon  system  are  performed.  This  could 
suppress  the  actual  cost  grow*  of  the  system  if  the  cost  variances  are 
compared  to  the  new  baseline.  The  reconstruction  of  the  data  into  the 
Slew  format  by  the  services  maintains  the  Milestone  II  baseline  as  its 
initial  estimate  and  reports  all  cost  variances  from  this  point.  This 
■should  allow  for  a  focus  on  all  forms  of  cost  growth  in  the  weapons 
system. 

E.  SSCAIATION  ADJUSTMENT 

The  data  used  is  in  constant  base  year  dollars.  The  base  year  for 
analysis  of  cost  growth  of  those  systems  that  are  beyond  Milestone  TI  is 
generally  chosen  as  the  year  the  project  enters  EMD.  Using  constant 
base  year  dollars  is  necessary  when  measuring  program  management  effects 
since  this  allows  for  consistart  comparisons  to  the  base  estimate  [Ref 
5] .  If  then  year  (budgetary)  dollars  were  used,  any  analysis  that 
combines  the  cost  variances  across  time  could  not  be  performed  since 
this  would  result  in  attempting  to  combine  different  year  dollars  which 
cannot  be  done.  Therefore,  for  this  analysis,  constant  base  year 
dollars  are  required. 

F.  QUANTITY  ADJUSTMENT 

Quantity  changes,  whether  a  change  in  the  total  quantity  procured  or 
a  change  in  the  procurement  rate,  will  greatly  affect  the  total  cost 
growth  of  a  system.  Milestone  II  cost  estimates  v/ere  based  on  the 
Milestone  II  quantity.  Any  subsequent  change  in  quantity  could  not  have 
been  foreseen  at  Milestone  II.  Therefore,  an  adjustment  to  the  cost 
variances  needs  to  be  made  tc  account  for  these  changes  in  qi.’antity. 

This  is  defined  as  quantity  normalization. 
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1.  Quantity  Related  Coat  Equation 

Before  developing  the  quantity  normalization  equation,  it  is 
necessary  to  understand  the  composition  of  the  current  estimate  (CE)  and 
hov/  it  relates  to  the  Milestone  II  decision  estimate  (DE) . 

Two  types  of  costs  exist  in  the  DE .  One  type  is  cost  not 
related  to  quantity.  These  are  costs  that  do  not  change  with  changes  in 
the  quantity  to  be  procured  or  changes  in  the  procurement  rate. 

Examples  of  these  costs  include  initial  design  engineering  and  tooling. 
The  total  nonquantity-related  cost  at  the  DE  is  defined  as  N. 

The  other  type  of  cost  in  the  DE  is  quantity  related.  The  sum 
of  the  quantity- related  costs  at  Milestone  II  is  defined  as  R.  These 
change  with  changes  in  quantity  or  changes  in  the  procurement  rate. 
Examples  of  these  costs  include  material  and  labor.  The  total  quantity- 
related  cost  at  Milestone  II  is  formed  by  the  product  of  the  Milestone 
II  unit  cost  (U)  times  the  Milestone  II  quantity  to  be  procured  (Q) .  By 
evaluating  the  reasons  given  in  the  SAR  for  each  of  the  cost  variances, 
any  cost  variance  listed  in  the  SAR  can  be  classified  as  be:  g  either 
quantity  related  or  not.  Therefore,  the  Milestone  II  DE  is  written  as: 

DE  =  N  +  R 
=  N  +  UQ 

The  total  change  in  the  nonquantity  related  costs  from  the 
decision  estimate  t.o  the  current  estimate  (AN)  can  be  calculated  by 
summing  all  of  the  cost  variances  that  are  not  quantity  related. 

The  total  change  in  the  quantity  related  costs  from  the  decision 
estimate  to  the  current  estimate  (AR)  can  be  calculated  by  summing  all 
of  the  cost  variance.^  that  are  quantity  related. 

The  current  estimate  (CE)  can  be  determined  from  the  DE  and  the 
cost  variances  by: 

CE  =  DE  +  AN  +  AR 
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A  quantity  related  cost  variance  can  be  caused  by  a  change  in 
the  quantity  to  be  procured  (AQ) ,  a  change  in  the  unit  cost  of  the  item 
(AU) ,  or  both.  Therefore,  CE  can  be  rewritten  as: 

CE  =  DE  +  AN  +  AR 

=  N  +  AN  +  R  +  AR 
=  N  +  AN  +  (U  +  AU)  (Q  +  AQ) 

=  N  +  AN  +  UQ  +  QAU  +  UAQ  +  AUAQ 

Ey  further  analyzing  the  reasons  for  the  cost  variances  listed 
in  the  SAR,  those  cost  variances  resulting  from  a  change  in  quantity 
with  no  change  in  unit  cost  can  be  identified.  The  cum  of  these  cost 
variances  is  defined  as  UAQ.  The  remaining  quantity  related  cost 
variances  are  due  to  changes  in  quantity  with  a  corresponding  change  in 
unit  cost  or  are  due  to  a  change  in  unit  cost  with  no  change  in  the 

quantity.  The  sum  of  these  cost  variances  is  defined  as  AU(Q  +  AQ) . 

Therefore,  CE  can  be  rewritten  as: 

CE  =  N  +  AN  +  UQ  +  QAU  4  UAQ  4  AUAQ 

=  (N  4  AN)  4  UQ  4  AU(Q  4  AQ)  4  UAQ 

Define  AP  as  the  difference  between  the  current  estimate  and  the 
decision  estimate: 

AP  =  CE  -  DE 

=  (N  4  AN)  4  UQ  4  AU(Q  4  AQ)  4  UAQ  -  (N  4  UQ) 

=  AN  H-  AU(Q  4  AQ)  4  AUAQ 

with  AN,  AU(Q  4  AQ)  ,  and  UAQ  each  being  a  determinable  qua.itity. 

2.  Quantity  Normalization  Equation 

The  quantity  normalization  equation  used  in  this  research  is 
currently  under  development  by  the  Office  of  the  Secretary  of  Defense. 

The  preferred  m.ethod  of  quantity  normalization  varies  widely 
[Refs.  4,  5] .  Two  types  of  effects  due  to  quantity  must  be  addressed. 
The  first  is  that  the  final  quantity  is  changed  with  no  change  in  tlie 
procurement  rate.  This  study  deals  with  the  total  costs  of  the  sy,3tem 
based  on  the  quantity  determined  at  Milestone  II  and  therefore  requires 


15 


that  this  type  of  quantity  change  not  result  in  a  cost  variance.  The 
second  type  of  qu.antity  effect  is  a  change  in  the  procurement  rate  with 
no  change  in  the  final  quantity.  Since  contractors  face  both  fixed  and 
variable  costs,  a  change  in  the  procurement  rate  such  as  a  schedule 
slippage  will  result  in  a  higher  unit  cost  since  additional  fixed  costs 
will  be  incurred.  This  type  of  cost  variance  must  be  fully  accounted 
for. 

The  data  collected  is  classified  as  to  whether  or  not  it  was 
quantity  related.  No  adjustment  for  gu.antity  is  required  if  it  is  not 
quantity  related.  The  quantity  adjustment  to  the  cost  variance  must  be 
made  to  the  base-year  adjusted  cost  variance  to  remove  any  inflationary 
effects . 

AP  is  not  quantity  nonnalized  since  a  change  in  the  quantity 
(AQ)  with  no  change  in  the  procurement  rate  (i.e.,  aU  =  0)  will  result 
in  a  change  in  AP. 

The  required  form  for  a  quantity  nornialized  pioouiernent  cost  C 
must  remove  any  effect  for  a  change  in  quantity  without  a  change  in 
procurement  rate  since  this  form  of  cost  variance  could  not  have  been 
foreseen  by  the  Milestone  II  estimator.s .  However,  C  must  fully  account 
for  procurement  rate  changes  since  these  are  attributable  to  decisions 
or  mi.stakes  that  should  have  been  foreseen  at  Milestone  II.  Therefore 
the  proper  form  for  C  is: 

C  =  (N  +  AN)  +  (U  -t-  AU)Q 

This  satisfies  the  requirements  li.sted  above,  namely  that 
changes  in  quantJ.ty  with  no  change  in  procurement  cost  are  igtiored  but 
that  changes  in  procurement  rate  which  affect  the  unit,  cost  are 
completely  accounted  for. 

Th.3  quantity  normalized  cost  variance  AC  is  the  quantity 
normalized  CE  (C)  minus  the  DE.  This  yields; 
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AC  =  C  -  DE 

=  (N  4-  AN)  +  (U  +  AU)Q  -  (N  +  UQ) 

=  an  +  QAU 

The  equation  for  determining  AC  can  be  obtained  as  follows ; 

AC  =  AN  +  QAU 

-  AN  +  QAU(Q  +  AQ)/(Q  +  AQ) 

The  terms  Q  and  AQ  can  be  taken  directly  'rom  the  EAR.  The 
remaining  terms,  AN  and  AU(Q  +  AQ)  can  be  determined  by  the  method 
listed  previously  when  determining  AP.  Therefore,  AC  is  the  quantity 
normalized  procurement  cost  variance  in  base  year  dollars.  AC  will  be 
used  in  the  analysis  as  the  quantity  normalized  cost  variance  to 
determine  the  cost  growth  of  the  weapon  system. 
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rv .  DATA 


Data  from  nine  weapon  Byaterng  are  currently  available.  In  oeveral 
yearn,  data  for  all.  oystemo  reported  under  the  SAR  guideline.'-,  will  be 
collected  and  all  of  the  data  will  be  analyzed  by  the  Office  of  the 
Secretary  of  Defenoe. 

Only  nine  syntemo  were  choBcn  due  to  the  limited  time  available  for 
data  collection  for  this  ntudy.  The  Department  of  DefenBe  uelected  the 
first  nine  nysterno  do  that  they  reflect  a  representative  oample  of  the 
ayntema  in  each  acrvice.  They  include  nhipo,  misailefi  and  electronic 
syur.emo.  No  aircraft  ayntems  v;ere  available,  and  it  should  be  noted 
that  the  results  of  this  analysis  should  not  be  applied  to  any  analysis 
of  aircraft  flystern.s.  There  are  approximately  equal  numbers  of  Army  and 
Navy  uystems .  Mo  Air  Korea  oystems  were  available. 


The  aie  uL  uifCoxeuw  in  thcii  uicinuin^  li£t:Li!uc  €Mid 

have  a  wide  range  of  totol  cootu  to  reflect  the  actual  nature  of  the 
overall  procurement  process  within  the  Department  of  Defenoe. 

Kour  of  the  syotemo  were  reported  by  the  Army.  They  were  the  ATACM 
misfiile,  the  MUtS  rocket  syotem,  the  SlNCGAR.'j  electronic  system  and  the 
MSK  electronic  system.  The  teinaining  five  oyutcniu  were  reported  by  the 
Navy.  Throe  arc  the  CH\MH-b3  helicopter,  the  Trident  II  submarine,  the 
DDCJ-51  .outface  ohip,  the  HARM  mintile  and  the  SM-2  mib.yile. 

The  data  for  each  weapon  system  was  conotrur.tcd  lioin  exintiny  :>AR 
data  h.y  the  respective  oervice .  The  data  consists  of  all  cost  variances 
reported  in  conotant  haoo  ycAt  ''ollars  from  the  SAH  sinct;  Milestone  II 
along  with  the  NAR  catcyor tzation ,  the  new  categorization,  cuid  whether 
or  not  the  coot  variance  is  quantity  related.  The  leported  data  also 


included  tlie  Milestone  II  RDTaK  and  proc.urcment  costi.  a;j  va.'ll  as  any 
amp.lilying  remarks  nec'-usaty  to  further  explain  any  cost  vai  iances  . 


A.  COST  GKOWTH  PATIO 


The  cofit:  growthi  ratio  (CGR)  is  defined  a£i  the  ratio  of  the  quantity 
and  escalation  adjusted  cost  variances  and  the  Milestone  II  total  syntem 
costs  including  RDTtE  costs: 

CGR  .  AC/DE. 

The  total  syutetn  cost,  is  used  so  that  the  CGR  represent.;!  the  change 
in  the  total  cost  of  the  system  as  opposed  to  a  percentage  change  in  the 
procurement  cost  of  the  system.  All  coat  variances  for  each  system  v/erc 
converted  to  CGRs .  The  CGR  is  used  in  nearly  ail  research  on  cost 
grov;th  to  show  the  magnitude  of  the  change  compared  to  the  original 
estimate.  I'or  instance,  a  fifty  million  dollar  cliange  in  a  system  that 
cost  two  h.undred  million  dollars  is  much  more  nignificanl  than  a  fifty 
million  dollar  cjiange  in  a  ten  billion  dollai'  oystom  since  the  first 
constitutes  a  relatively  large  change  in  the  accuracy  of  the  original 
estimate  (CGR  ■  0.2H),  wliile  the  latter  is  well  wi* hin  any  reasonable 
expectations  for  margin  of  eatii;«cir.g  error  (CGR  «  O.OOS). 

For  each  oyotem,  the  CGP.o  associated  with  a  particular  category, 
using  both  chc  SAR  and  the  new  catogori-ration  systems,  were  aggregated 
to  form  the  cost  growth  for  a  system  classified  by  category.  The  CGRs 
Lliat  arc  associated  with  t.hc  mistake  catcgoiies  were  aggregated  to  tons 
the  mistake  cost  growth  and  iti  a  similar  manner  the  dcci-iion  categories 
were  comijiried  to  form  tlie  decision  cost  growth. 

The  sum  of  the  decision  and  mistake  cost  growths  form  the  total  cost 
growth  of  a  tyutern  since  Milestone  II. 

B.  bOALiAR  tfBZGaTELi  AVERAGE 

When  systemn  are  comiiined  into  sets  tor  analysis  ol  the  effect;!  of 
common  factora,  the  dollar  weighted  average  is  used  a.i  the  mean  value 
instead  of  the  arithmetic  mean.  This  is  d<jne  to  account  toi  tin*  gieaier 
imyjacl  that  cost  growth  of  mote  ex)jeiisive  weap.on  systems  ha.s  on 
budgetary  coiuideiationo .  The;  objective  or  tliis  resena  '  ch  is  to 
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determine  what  factors  are  causing  cost  growth  within  the  entire 
Department  of  Defense  rather  than  on  a  particular  system.  This  type  of 
analysis  s)iould  aid  in  determining  the  impact  of  cost  growth  cn  the 
military's  procurement  budget.  A  strictly  arithi.ietic  average  is  not 
appropriate  in  a  budgetary  context  .since  the  effect  of  less  expensive 
systems  would  skew  the  combined  CGRs  away  from  the  expected  cost  growth 
in  the  procurement  of  all  weapons  .systems. 

The  weapon  systems  will  be  divided  into  various  sets  in  the  analysis 
of  their  cost  growth  (e.g..  Army  and  Davy  systems).  The  dollar  weighted 
average  (DW/t)  for  a  .net  of  weapon  sy.sterns  is  determined  by: 

DWA  -  DiW,(CGHi)  i  (  {all  sv'stema  in  the  set} 

where  w,  -  TC,/  (biTC,)  Vi  t  (all  systems  in  the  set} 

LiW  »  1  it  (all  systems  in  the  set} 

'i'C^  ■  Milestone  l.i  total  system  cost  of  system  i. 

The  TCjS  for  Die  weapon  systems  in  the  set  must  all  be  converted  to  the 
same  year  dollars.  For  this  analysie,  all  i'ilectone  II  sotimateo  were 
converted  to  lSf93  constant  dollars  using  tlie  OSD/DAVCOMPT  Guidance  of 
March  1993  doted  2C  Marcli  19!>3. 

C.  SIHCOARS  SV8TEK  ADOMALY  AND  TREATMKNT 

The  Army's  SIDCGAHS  electronic  system  lias  a  CGH  (if 
-0.777  tor  tlie  oevent.li  year  since  Milestone  II,  This  results  f  roni  a 
decision  in  that  year  for  a  large  scale  reduction  in  tlie  syutem.  T)iis 
decision  is  not  typical  of  most  weapons  uystei  s .  Tlv:  data  lor  the  first 
six  years  is  typical  and  will  be  used.  Tlie  first  six  yeai  i;  since 
Milestone  II  had  a  total  CGH  of  0.10b. 

D.  TOTAL  DATA  SET 

Table  I  diopldyn  tlsi  MLleiilone  J1  year,  the  Milestone  11  decision 
estimate  (UK)  in  millions  of  base  year  dollais,  and  the  iium)je,  of  years 
of  data  since  Milestone  II  available.  Kadi  d.ita  sei  les  extends 

only  tliiougli  Idle  1  inal  yeai  re)JOited  by  eadi  seivjct;. 
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Table  2  displays  the  unweighted  cost  growth  ratios  for  the  decision 
categories  and  the.  total  decision  cost  growth  ratic  for  each  system.  It 
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also  display.s  the  dollar  weighted  average  of  each  category. 

Table  3  displays  the  unweighted  cost  growth  ratios  for  the  mistake 
categories  and  the  total  mistake  cost  growth  ratio  for  each  system.  It 
also  displays  the  dollar  weighted  average  of  each  category. 

Table  4  displays  the  unweighted  decision,  mistake,  and  total  cost 
growth  ratios  for  each  system  and  the  dollar  weighted  average  of  each 
cat^egory . 


TABLE  1 .  BASELINE  SYSTEM  DATA 


System 

Milestone  II  Year 

DE  (in  MilliciiB  of 
Base  Year  Dollars) 

Years  of 
Data  Used 

ATACM 

1990 

1506 . 0 

2 

CJ:\KM-Ji3 

1979 

510 . 0 

12 

DDG-bl 

1987 

16723 . 8 

4 

HARM 

1982 

3274 . 3 

10 

HLRS 

1984 

2077 . R 

7 

1989 

3540 . 4 

3 

SIKCGARS 

1983 

4145 . 0 

G 

bM-2 

1984 

6571 . 5 

8 

Trident 

1906 

142S8 . 9 

4 

TABLE  2.  UNWEIGHTEU  COBS  FOR  DBCI.’^IOIJ  CATKOORIES 


Derv 

Usmmi 

Dilu 

mm 

Dotlier 

Decision 

ATACM 

0 

0 . 023 

0 

0.015 

0 

0 

0 . 030 

C’H\M11  -  53 

-0 . 009 

0.018 

-0.310 

0 

0 

0 

-0.301 

DDU- 91 

0.041 

0  .  100 

0.009 

0 

0 

0 . 196 

HAJUl 

-0 . 142 

0.298 

0 . 015 

0 . 03  0 

■HB 

0 

0.161 

MLKS 

0 . 079 

0.014 

0.0.39 

'0 . 008 

fj 

0 

0.120 

MSE 

0 

0 . 018 

0 

0 

0 

0 

0 . 018 

.‘JINCCiAkS 

0.156 

0 

-0 . 037 

0 

0 

0.119 

St4-2 

0 . 016 

0 . 072 

0.010 

-0.029 

0 

0 

0,073 

Trident 

-  0 . 022 

0 

0 

0 

0 

0 

-0.022 

Average 

(J  .  0 1  9 

0 , 061 

-  0.001 

-0.001 

0 . 002 

0 

0 . 07  5 

2  I 


TABLE  3  .  tnrtBIOHTBD  C09.B  FOR  MISTAKE  CATKQORIKS 


Mcep 

Mcede 

Mil.': 

Msarr.f 

Mescl 

Mother 

Mistake 

ATACM 

0 

0 

-0. oil 

0 

0.020 

0 

0 . 009 

CH\t-lll-53 

0.529 

0 

0.047 

0 

0 

0 

0 . 57C 

DDG-51 

0 . 113 

0 

-0 . 005 

0 

0 

0 

0 . 108 

HARM 

0 . 037 

0 

0 . 022 

-0.005 

0 . 002 

0 

0 . 056 

MLRS 

-0.036 

0 

-0 . 006 

0 

0 

-0 . 020 

-0 , 062 

CO 

0 . 082 

-0 . 010 

0 

0.003 

0 

0 

0.075 

SIHCGARS 

-0.030 

0 

0 . 005 

0.011 

0 

0 

-0 . 014 

SH-?. 

-0.033 

0 

-0 . 009 

0 

-0 . 014 

0 

-0 . 056 

-0.039 

-0 . 001 

0 

0 

0 . 014 

-0.024 

Average 

0.031 

-0 . 001 

-0 . 001 

0.001 

O 

O 

O 

0 

0 . 038 

TABLE  4 .  UNWEIGHTED  SYSTEM  COST  GROWTH  RATIOS 


Decioion 

Mistake 

Total 

ATACM 

0.038 

0.009 

0 . 047 

CH\MK- S3 

-0.301 

0 . 576 

0 . 276 

DDG -  5 1 

0.156 

0  .  i08 

0,264 

HARK 

0, 161 

0 . 056 

0.217 

MLRS 

o 

o 

-0.062 

0 . 058 

MSE  0,018 _ 0.075 _ 0,092 

INCGAJES  0.119  -0.014  0.105 


SM-2 

0 . 073 

-0 . 056 

0 . 017 

Trident 

-0 . 022 

-C  .024 

-0 , 046 

Aver,  ago 


0 . 075 


0.033 


0 , 100 


V.  tBTHODOtOOY  AMD  KMSULT3 


A.  USBl^aiiMKSS  OF  HtW  CATXOORIK8 

In  oxrdar  co  more  readily  identify  areas  of  cost  growth,  it  is 
necessary  to  classify  the  cost  variances  into  a  more  useable  and  more 
detailed  categorization  system.  The  new  system  should  classify  the  cost 
growths  into  more  categories  than  the  SAR  categorization  method. 

This  system  shows  data  counts,  and  in  would  be  inappropriate  to  use 
analysis  of  variance  (AKOVA)  since  Che  data  is  not  normally  distributed. 
However,  a  simple  comparison  of  the  number  of  categories  needed  under 
each  system  will  suffice  in  this  determination. 

Tabic  5  displays  the  number  of  SAR  categories  aa  compared  to  the 
number  of  new  categories  used  to  classify  each  system's  cost  variances 
since  Milestone  II 

As  can  be  seen  from  the  table,  all  systems  require  a  larger  number 
of  categories  to  classify  the  cost  variances  using  the  new  system  than 
using  the  SAW  system.  Therefore,  those  factors  having  the  greatest 
impact  on  cost  growth  can  ba  more  readily  identified  and  the  underlying 
causes  of  each  of  the  most  significant  contributors  can  be  determined. 

TABLE  S .  WXrUBBR  OF  RBgPlHEO  CATBOORICB 


B.  STATISTICAL  APPROACH 

1.  R*gi:«f(8ion  And  Tim*  Sarlas  Modala 

A  regression  model  of  the  cost  growth  ratio  would  appear  to  be 
useful  when  analyzing  the  cost  growth  of  weapon  systems.  However,  with 
the  limited  data  available,  combined  with  the  large  variation  in  CGRs, 
the  only  regression  model  that  fits  is  the  full  or  saturated  model  which 
does  not  provide  any  beneficial  information.  Also,  since  the  objective 
of  this  research  is  not  to  predict  the  cost  growth  of  a  specific  system 
but  inetead  to  describe  the  cost  growth  of  the  entire  Department  of 
Defense,  a  regression  model  would  not  be  of  use. 

A  time  series  approach  modeling  the  cost  growth  versus  time  of 
weapon  systems  is  also  of  little  benefit.  The  uniqueness  of  an 
individual  weapon  syttem  and  the  dociaiona  that  affect  a  particular 
aystam  do  not  allow  for  general  system  models  of  cost  growth  for  these 

weapon  ayetemii . 

2  .  Data  Groups 

The  Weapon  systenio  will  be  broken  into  seLs  to  determine  if  a 
specific  attribute  of  each  set  of  syatems  has  an  impact  on  the  cost 
growth  of  the  set.  The  following  groupings  or  sets  of  weapons  systems 
will  be  used  to  analyze  the  data:  low  cost  versus  high  cost  systems, 
missile  systems  versus  nonmisuile  systems,  and  Army  versus  Navy  systems. 
An  analysis  of  tho-ne  categories  whore  a  majority  of  the  cost  growth 
occurs  in  each  of  these  sets  can  be  used  to  evaluate  the  causes  of  cost 
growth.  The  reasons  for  each  of  the  groupings  will  be  discussed  in 

detail  in  each  of  the  sections  below. 

3 .  Tasting  Fox  Comon  Naana 

The  assumption  of  normality  of  the  cost  growth  ratios  in  ordei- 
to  test  for  common  means  using  parametric  tests  would  be  'difficult  to 
make  since  the  CGK  cannot  go  below  a  value  of  -1.0.  The  appropriate 
rionparametric  test  for  conuiion  means  ia  the  V/ilcoxon  Rank  Sum  Te.gt  which 
is  equivalent  to  the  Mann-Whitnoy  U  Test.  The  value  of  the  Wilcoxon 
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test  statistic  (w)  will  be  given  along  with  the  associated  level  of 
significance  (p  value) . 

If  the  p  value  associated  with  the  Wilcoxon  test  is  not 
determinable  as  significant  or  not  (i.e.,  between  0.1  and  0.15)  and 
paired  system  data  is  being  evaluated,  the  Friedman  nonparametric  test 
will  be  used  to  determine  whether  or  not  to  accept  the  null  hypothesis. 
The  Friedman  chi -squared  value  (X*)  will  be  given  along  with  the 
associated  level  of  significance  (p  value)  whe.n  this  test  is  conducted. 
[Ref  o] 

a.  Dollar  Weighting  the  Data 

The  mistake,  decision,  and  total  CGRs  of  each  element  i  in  a 
set  need  to  be  transformed  to  account  for  the  different  total  cost  each 
system  has.  This  allows  for  a  larger  weight  to  be  placed  on  the  CGR  of 
the  higher  cost  systems  in  the  set  and  a  lesser  weight  to  be  placed  on 
the  less  expensive  systems  in  the  set.  Each  CGR  will  be  adjusted  using 
a  dollar  weighting  factor.  The  dollar  weighting  factor  (w^)  is 
determined  for  eacli  element  i  of  a  set  of  weapon  systems  and  is 
equivalent  to  the  defined  in  calculating  the  dollar  weighted  average. 
Before  a  teat  for  common  means  is  performed,  the  mista)ce,  decision,  and 
total  CGRs  of  each  element  i  in  the  set  is  multiplied  by  its  dollar 
v/eighting  factor  Wf  The  dollar  weighted  CGRs  are  then  used  for  the 
common  means  tests . 

C.  DECISION  VERSUS  MISTAKB  COST  GROWTH 

An  analysis  of  the  difference  between  decision  cost  grov;th  and 
mistake  cost  growth  of  all  of  the  system.^  is  recfuired  to  determine  how 
accurate  Milestone  II  estimates  are  and  to  .identify  those  areas  where 

improvements  in  our  estimating  techniques  need  to  be  t  de. 

1.  H/pothesln 

The  Wilcoxon  Rank  Sum  Test  is  u.sed  to  test  the  null  hypothe.sis 
that  the  mean  cost  growth  due  to  decisions  is  equal  to  the  mean  cost 
growth  due  to  mistakes.  The  alternative  hypothesis  is  that  the  mean 
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cost  growth  for  decisions  is  greater  than  the  mean  cost  growth  due  to 
mistakes.  The  common  means  test  is  performed  on  the  dollar  weighted 

data  for  all  nine  weapon  systems. 

2 .  Data 

Table  6  displays  the  dollar  weighted  decision,  mistake  and  total 
cost  growths  for  all  the  weapon  systems. 


TABLE  6.  COST  GHOWTH  RATIOS  FOR  EACH  SYSTEM 


System 

Decision 

Mistake 

- - - 

Total 

ATACM 

0.001 

0.000 

0 . 001 

CH\MH-53 

-0 . 004 

0 . 008 

0 . 004 

DDG-51 

0 . 048 

0 . 033 

0 . 081 

HARM 

0.011 

6 

o 

o 

0 . 015 

MLRS 

O.OOS 

-0.003 

0 . 002 

MSE 

0.001 

0 . 004 

0 . 006 

SINCGARS 

0.010 

-0.001 

0 . 009 

£M  •'  2 

0 . 010 

- 0 . 007 

A  AAA 

W  .  V  w  A. 

Trident  II 

-0.006 

-0 . 006 

-0.012 

3.  Results  o£  Decision  Versus  Mistake  Cost  Growth 

The  Wilcoxon  rank  sum  test  for  common  means  for  decision  and 
mistake  cost  growth  yields  a  test  statistic  value  of  W  =  72  with  a  p 
value  of  0.13.  This  may  or  may  not  be  significant,  so  a  Friedman  test 
was  performed  and  yields  a  test  statistic  value  of  =  2.778  with  a  p 
value  of  0.096.  This  is  significant  and  the  null  hypothesis  of  equal 
means  is  rejected.  Therefore,  the  mean  decision  cost  growth  is  greater 
than  the  mean  mistake  cost  growth.  The  dollar  weighted  average  (DWA) 

decision  cost  growth  is  0.075  and  the  DWA  mistake  cost  growth  is  0.033. 

4.  Categorical  Analysis 

Table  7  displays  the  dollar  weighted  average  CGRs  of  the 
decision  and  mistake  categories.  No  statistical  tests  were  performed  to 
determine  which  categories  contain  the  largest  cost  growth;  only  a 
visual  inspectio.u  is  required. 
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TABLE  7.  AVERAGE  CATEGORY  CORS  FOR  ALL  SYSTEMS 


Decision 

Dcrv 

Dsmmi 

Dils 

Depf 

Descl 

Dother 

CGR 

0.015 

0 . 061 

-0.001 

-0  .  OOl 

0 . 002 

0 

Mistake 

Mcep 

Mcede 

Mils 

Mssmf 

Mescl 

Mother 

CGR 

0.031 

-0 . 001 

-0.001 

0.001 

0.003 

0 

The  majority  of  decision  cost  growth  occurs  in  the  Dsmmi 
category  and  has  a  dollar  weighted  average  value  of  0.061.  The  majority 
of  mistake  cost  growth  occLts  in  the  Mcep  category  and  has  a  dollar 
weighted  average  value  of  0.031. 

D.  LOW  COST  VERSUS  HIGH  COST  SYSTEMS 

A  determination  of  the  relationship  between  system  cost  and  system 
cost  growth  is  needed  to  determine  if  a  difference  exists  between  the 
cost  growth  of  low  cost  systems  and  the  cost  growth  of  high  cost 
systems.  A  low  cost  system  is  defined  as  any  system  whose  total  system 
cost  estimated  at  Milestone  TT  is  less  than  three  and  a  half  billion 
dollars.  The  value  of  three  and  a  half  billion  dollars  is  somewhat 
arbitrary,  but  this  number  was  chosen  to  split  the  number  of  systems  in 
each  grouping  approximately  in  half  and  to  ensure  that  an  approximately 
equal  ratio  of  Army  to  Navy  systems  exist  in  each  set  to  eliminate  any 
possible  service  effects.  The  low  cost  systems  are  the  ATACM,  the 
CH\mH-53,  the  ILARM  and  the  MLRS .  The  systems  in  the  high  cost  category 

are  the  DDG-51,  the  MSE,  the  SINCGARS,  the  SM-2  and  the  Trident  II. 

1.  Hypothesis 

The  Wilcoxon  Rank  Sum  Test  is  used  to  test  the  null  hypothesis 
that  the  mean  cost  growth  of  low  cost  systems  is  equal  to  the  mean  cost 
growth  of  high  cost  systems.  The  alternative  hypothesis  is  that  Che  low 

cost  systems  have  a  higher  mean  cost  growth  than  the  high  cost  systems. 

2 .  Data 

Table  8  displays  the  dollar  weighted  cost  growth  ratio.s  for  the 
low  cost  weapon  systems . 
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Table  9  displays  the  dollar  weighted  cost  growth  ratios  for  the 


high  cost  weapon  systems. 

TABLE  8.  DOLLAR  WEIGHTED  CGRS  FOR  LOW  COST  SYSTEMS 


.System 

Decision 

Mistake 

Total 

ATACM 

0 . 006 

0.001 

0 . 008 

CH\MIi-53 

-0.029 

o 

o 

Ln 

0 . 026 

HARM 

0.075 

0 . 026 

0 . 102 

MLRS 

0.033 

-0.017 

0 . 016 

lABLE  9.  DOLLAR  WEIGHTED  CGRS  FOR  HIGH  COST  SYSTEMS 


System 

Decision 

Mistake 

Total 

DDG-51 

0.056 

0 . 039 

0 . 095 

MSE 

0 . 001 

0.005 

0 . 007 

SINCGARS 

0,012 

-0.001 

0  .  Oil 

SM-2 

O.OM 

-0.009 

0 . 003 

Trident  II 

-0.007 

-0.008 

-0.014 

3 .  Couqparlson  between  High  Cost  and  Low  Cost  Systems 
a,  2'otai  Cost  Growth 

The  wilcoxon  rank  sum  test  for  common  means  for  total  cost 
growth  of  low  cost  and  high  cost  systems  yields  a  test  .statistic  value 
of  w  =  19  with  a  p  value  of  0.08.  This  is  significant  and  the  null 
hypothesis  of  equal  means  is  rejected.  Therefore,  the  mean  total  cost 
growth  of  low  coat  systems  is  greater  than  the  mean  total  cost  growth  of 
high  cost  systems.  The  DWA  total  cost  growth  for  low  cost  systems  is 

0.152  and  the  DWA  total  cost  growth  for  high  cost  systems  is  0.100. 
h.  Decision  Cost  Growth  Comparison 

The  wilcoxon  rank  sum  test  for  common  means  for  decision 
cost  grov/th  of  low  cost  and  high  cost  systems  yields  a  test  statistic 
value  of  w  =  22  with  a  p  value  of  0.311.  Tliis  is  not  significant  and 
the  null  hypothesis  of  equal  means  is  not  rejected. 
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c.  MisCaJce  Coat  Gcowth  Comparison 

The  Wilcoxon  rank  sum  test  for  common  means  for  mistake  cost 
growth  of  low  cost  and  high  cost  systems  yields  a  test  statistic  value 
of  VI  =  21  with  a  p  value  of  0.35.  This  is  not  significant  and  the  null 
hypothesis  of  equal  means  is  not  rejected. 

The  difference  between  the  mistake  cost  growths,  while  not 
statistically  different,  is  substantively  different.  The  total  cost 
growth  of  low  cost  systems  is  greater  than  that  of  high  cost  systems, 
and  it  can  be  observed  that  the  difference  between  the  mistake  cost 
growths  is  the  driving  factor  behind  this  difference.  The  DVIA  mistake 
cost  growth  for  low  co.st  systems  is  0.066,  v/hile  the  DWA  total  cost 

growth  for  high  cost  systems  is  0.027. 

4.  bow  Coat  Systems 

a.  Decision  Versus  Mistake  Coat  Growth 

The  Wilcoxon  rank  sum  test  for  common  means  for  decision 
cost  growth  and  mistake  cost  growth  of  low  cost  systems  yields  a  tc.st 
statistic  value  of  W  =  17  with  a  p  value  of  0,38.  This  is  not 

significant  and  the  null  hypothesis  of  equax  means  is  not  rejected. 

b.  Categorical  Analysis 

Table  10  displays  the  dollar  weighted  average  CGRs  of  the 
decision  and  mistake  categories  for  low  cost  systems.  No  statistical 
tests  were  performed  to  determine  which  categories  contain  the  largest 
cost  growth;  only  a  visual  in.spection  is  required. 

TABLE  10.  AVERAGE  CATEGORY  CGRS  FOR  LOW  COST  SYSTEMS 


Decision 

Dcrv 

Dsmmi 

Dils 

Depf 

Descl 

Bother 

CGR 

-0 . 046 

0.130 

-0.013 

0 . 014 

0 

0 

Mistake 

Mcep 

Mcede 

Mils 

Mssmf 

Mescl 

Mother 

CGR 

0.058 

0 

0.011 

-0 . 002 

0.004 

-0.005 

The  majority  of  decision  cost  growth  for  low  cost  systems 
occurs  in  the  Dsmmi  category  and  has  a  dollar  weighted  average  value  of 


0.130.  The;  majority  of  mistake  cost  growth  in  lov;  cost  systems  occurs 

in  the  Mcep  category  and  has  a  dollar  weighted  average  value  0.058. 

5.  High  Coat  Syatama 

a.  Decision  Versus  Mistake  Coat  Growth 

The  wilcoxon  rank  sum  test  for  common  means  for  decision 
cost  growth  and  mistake  cost  growth  of  high  cost  systems  yields  a  test 
statistic  value  of  W  =  22  with  a  p  value  of  0.1-1.  This  may  or  may  not 
be  significant,  so  a  Friedman  test  was  performed  and  yields  a  test 
statistic  value  of  =  1.8  with  a  p  value  of  0.18.  This  is  not 
significant  and  the  null  hypothesis  of  equal  means  is  not  rejected. 

The  difference  between  the  mistake  and  deci.sion  cost 
growths,  while  not  stati  .stically  different,  is  substantively  different 
when  compared  to  the  difference  between  the  decision  and.  mistake  cost 
growths  of  lov;  cost  system.s .  The  DV7A  mistake  cost  growth  for  high  cost 
systems  0.027  and  the  DWA  decision  cost  growth  for  high  cost  systems  is 
0 . 0 7-1 . 

L.  Categorical  Analysis 

Table  11  displays  the  dollar  weighted  average  CGRs  of  the 
decision  and  mistake  categories  for  high  cost  systems.  No  statistical 
teats  were  performed  to  determine  which  categories  contain  the  large.su 
cost  growth;  only  a  visual  inspection  is  rcqiiircd. 


TABLE  11.  AVERAGE  CATEGORY  CORS  FOR  HIGH  COST  SYSTEMS 


Decision 

Derv 

Usmrni 

Oils 

Depf 

Dother 

CGR 

0 . 026 

0,048 

0 . 001 

-0,004 

0 . 002 

0 

Mistake 

Mcep 

Mcede 

Mils 

Mesmf 

Mend 

Mother 

CGR 

0.026 

-0 . 001 

-0.003 

0 . 001 

0 . 002 

0.001 

The  majority  of  deci.>3ion  cost  grov/th  for  high  cost  systems 
occurs  in  the  Dsmrr.i  category  and  has  a  dollar  weighted  average  value  of 
0,048.  The  majority  of  mistake  cost  growth  for  high  co.st  sy.stem.s  occurs 
in  the  Mcep  category  and  has  a  dollar  wr ighted  average  value  of  0,026. 


K.  MISSILE  SYSTEMS  VERSUS  OTHER  SYSTEMS 


An  analysis  of  the  cost  growth  of  missile  systems  compared  to  other 
weapons  systems  is  beneficial  since  it  will  allow  for  identification  of 
differences  in  the  procurement  and  estimation  processes  of  these  systems 
that  may  need  to  be  evaluated  in  further  detail  in  future  studies,  Tlie 
missile  systems  consist  of  the  ATACM,  the  HARJ"1  and  the  SM-2.  These  two 
sets  have  approximately  eq^jal  low  cost  to  high  cost  ratios  and  Army  to 
Navy  system  ratios.  This  should  remove  any  effects  that  system  cost 
would  have  on  the  data.  The  data  was  then  dollar  weighted  within  each 
set . 

1.  Kypothcais 

The  Wilcoxon  Rank  .Sum  Test  is  performed  to  test  the  null 
hypothesis  that  the  mean  co.st  growt  i  of  miusile  systems  is  equal  to  the 
mean  cost  growth  of  other  weapon  systems.  The  alternative  hypothesis  is 
that  the  missile  systems  have  a  lower  mean  cost  growth  than  the  other 

weapon  systems. 

2.  Data 

Table  12  display.^  the  dollar  weighted  cost  growths  for  missile 

systems . 

Table  13  displays  the  dollar  weighted  cost  growth.^  for  the 
nonrnissile  systems. 


TABLE  12.  DOLLAR  WEIGHTED  CGRS  FOR  MISSILE  SYSTEMS 


System 

Decision 

[Mistake 

Total 

ATACM 

0 . 004 

0 . 001 

0 . 005 

H/U?M 

0 . 050 

0.018 

0.068 

SM-2 

0.042 

-0 , 032 

0.010 
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TADLl  13.  DOLLAR  WBIOHTKn  CORa  FOR  NONMISSILE  SYSTEMS 


System 

Decision 

Mistake 

Total 

CH\MH-53 

-0 . 006 

0.011 

0 . 005 

DUG- 51 

0 . 062 

0.043 

0 . 104 

MSE 

0.006 

-0 . 003 

0.003 

SINCGARS 

C  001 

0 . 006 

0 . 007 

SM-?. 

0.013 

-0 . 002 

0 . 012 

Trident  II 

-0.008 

-0 . 008 

-0.016 

3.  Coiiqparison  betw««n  Miasil*  and  NonsilBBlla  Syatama 

a.  Total  Coat  Grot'ftb 

The  Wilcoxon  rank  sum  test  tor  common  means  for  total  cost 
growth  c£  missile  and  nonmiasile  systems  yields  a  teet  statistic  value 
of  W  =  17.5  with  a  p  value  of  0.27.  This  is  not  significant  and  the 

null  hypothe.sis  ot  equal  mean.s  is  not  rejected. 

b.  Declfllon  Coat  Growth  Compariaon 

The  VJilcoxon  rank  sum  test  for  common  means  lor  decision 
cost  growth  of  missile  and  nonmissile  systems  yields  a  test  statistic 
value  of  W  »  19  with  a  p  value  of  C .  16 .  This  is  not  significant  and  the 

null  hypothesis  of  equal  means  is  not  rejected. 

c.  Hiataka  Coat  Growth  Compariaon 

The  wilcoxon  rank  sum  test  for  common  means  for  mistake  cost 
growth  of  missile  and  nonmissile  systems  yields  a  test  statistic  value 
of  W  =  14  with  a  p  value  of  0.40.  This  is  not  significant  and  the  null 
hypothe.sis  of  equal  means  i.s  not  rejected. 

The  di.ffr^rence  between  the  mistake  cost  growths  for  missile 
and  nonmissile  systems,  while  net  statistically  different,  is 
substantively  different.  It  can  be  readily  observed  that  tiie  difference 
between  the  mistake  cost  growths  ia  relevant  when  compared  to  the 
decision  cost  grov;ths  as  is  discussed  below.  The  DWA  mistake  cost 
growth  for  missile  systems  is  -0.014  and  the  DWA  total  cost  growtbi  for 
nonmissile  systems  is  0.046. 
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4.  Klacil*  SyatMUi 

a.  DeclB.ion  Voraua  Miattike  Coat  Growth 

The  Wilcoxon  rank  sum  tenc.  for  common  means  for  decision 
cost  growth  and  mistake  corit  growth  of  tninsile  nystems  yields  a  test 
stctifitic  value  of  W  »  7  with  a  p  value  of  0.076.  This  is  significant 
and  t)ie  null  hypothesis  of  equal  means  is  rejected.  The  DWA  mistake 
cost  growth  for  missile  systems  -0.014  and  the  DWA  decisiori  cost  growth 

for  missile  systems  is  0.097. 

h.  Categorical  Analyaia 

Table  14  displays  the  dollar  v/eighted  average  CGP.s  of  tlie 
decision  and  mistake  categories  for  miBHilc  nyntemu.  No  statistical 
teste  were  performed  to  determine  v/hich  categories  contain  the  largest 
cost  grov/th;  only  a  visual  inspection  i.s  required. 


TABLE  14.  AVERAGE  CATEGORY  CQRS  FOR  MISHILB  SYSTEMS 


Decision 

Derv 

Oils 

Depf 

Deuel 

III  .  -  ^ 

Dother 

CGR 

-0.035 

0 . 125 

0 . 010 

-0.003 

0 

0 

Mistake 

Keep 

MCede 

Mils 

Mfluiiif 

Meucl 

Mother 
_ 1 

CGR 

-0 . 007 

0 

0 . 001 

-0 . 002 

-0  005 

o 

) 

1 

The  majority  of  decision  cost  growtli  for  missile  systems 
occurs  i.n  the  Dammi  category  and  has  a  dollar  weighted  average  value  of 
0.12b.  The  majority  of  mistake  coat  grov/th  for  mi.'jaile  .sy.stems  occurs 

in  the  Keep  category  and  has  a  dollar  weighted  average  value  of  -0.007. 

5.  Nonulsalla  Sysk«mB 

a.  Decision  Veraus  Mistake  Coat  Growth 

The  Wilcoxon  rank  sum  test  for  common  means  for  decision 
cost  growth  and  mistake  cost  growth  of  nonmissile  systems  yield.s  a  test 
statistic  value  of  W  =  30,  with  a  p  value  of  0.4.3.  This  is  not 

significant  and  the  null  hypothesis  of  equal  means  is  not  rejected. 

Jb.  Categorical  Analysis 

Table  lb  displays  the  dollar  weighted  average  CGRs  of  the 
decisiori  and  mistake  categories  for  nonmissilc  systems.  No  statistical 
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toots  v/ero  j)ertormed  to  detoimino  which  categories  cor.tain  the  largest 
cost  growth;  only  a  ^'loual  inspection  is  required. 


TAJDLE  15.  AVHRA,OK  CATEGORY  CGRS  FOR  KONMISSILE  SYSTEMS 


Decision 

Dct  V 

Dsmmi 

Oils 

Depf 

1 -  ' 

Descl 

Dot her 

CGR 

0 . 030 

0.042 

-0.005 

0 

0 . 002 

0 

Mistake 

Mcep 

Mcede 

Mils 

Hssrrif 

Mcscl 

Mother 

CGR 

0.042 

-0.001 

-0.001 

0 , 001 

o 

o 

o 

0 

The  inajoiity  of  decision  cost,  growth  for  nonmissile  systems 
occurs  in  tv;o  categories,  the  Dafroni  category  and  the  Derv  category,  The 
Dsmmi  category  has  a  dollar  weighted  average  value  oi  0.042.  The  Dcj v 
category  hau  a  dollar  weighted  .iveiage  value  of  0.030.  "he  r  ajority  of 
mistake  cost  growth  for  rion  miooile  syotenio  occurs  in  th.o  Mcep  category 
and  l.ao  a  dollar  weiglitcd  average  value  of  0.042. 

r.  AitMY  VERSUS  NAVY  SYSTEMS 


^  i.  nf  C«4 


I _ _  i  _ _ I 


to  determine  which  service,  if  either,  lias  a  bettor  eotjmai.ion  or 

decision  making  process, 

1.  Kypotheala 

The  WJ  Icoxon  Rank  Sum  Test  is  performed  to  toot  tl)<;  null 
hypothusia  that  the  ntcan  cost  growth  of  Aritiy  systemv  is  equal  to  this 
mean  cost  growth  of  Uavy  systems.  The  alter  jiative  hypothesis  is  that 

the  means  are  tiot  equal  . 

2.  Data 

Table  16  displays  the  dollar  weighted  cor  growtlis  tor  Army 
systems  along  with  the  dollar  weighted  average  (DKA)  of  these  systems. 

Table  IV  displ.iy.'j  the  dollar  veighteri  coot  g/'>wth:i  for  (he  (.'.»vy 
systems  along  with  the  dollar  wi  igliled  average  (DWA)  of  these  flystemo. 


VXBLB  16.  DOLLAR  KBIQltTBD  COKS  FOR  ARMY  SYSTEMS 


System 

Decision 

Mistake 

Total 

ATACM 

0 . 004 

0 . 001 

0  .  OO'j 

MLR  .9 

0 . 02.3 

-0.012 

0  .  Oil 

MSh 

0.  OOS 

0 . 021 

0 , 02G 

SIMCGARS 

0.048 

-0 . 006 

0,043 

TABLE  17  ,  DOLt^  WHIQHTKD  C(JRS  FOR  NAVY  SYSTEMS 


Syutem 

Deciuion 

Mistake 

Total 

CU\MiPb3 

-0 , OOC 

0.011 

0.005 

DDG - S 1 

0 . 060 

0.042 

0 . 102 

HARM 

0 . 014 

7 . 00b 

0.020 

SM  -2 

0.012 

-0 . 009 

0.003 

Trident  II 

-0 . 007 

-  0 . 008 

O 

o 

I 

3.  Coia(}»xjoon  between  Amy  end  Nevy  Syeteme 
«.  Total  Coat  GiowUi 

Tho  'Wilcoxoii  tank  Hurn  tcMU  £oi  ooii  lof)  meano  lot  total  cout 
yrowth  ol  Army  and  Navy  (iy(jt«mH  ylcldo  a  tent  atatiotic  value  ol  w  > 

21. S  with  n  p  value  ol  0.21.  Thla  iu  nut  ulynK leant  and  the  null 

hypoihtaia  ot  equal  muauH  iu  nut  rojectad. 

Jb.  Leciaion  Coat  Crowth  Comi)»t.i0on 

The  Wllcoxon  tank  wui.i  tout  for  cummon  tnoaini  lor  docinion 
coHt  yiowlh  ot  Army  and  Navy  uyntomu  yto.ld(i  a  tcBt  otatiwtic  value?  of  Vi 
-  22  with  a  p  value  cjt  0.31.  Thiii  iu  nut  iiiyinlicant  and  the  n>jll 

hypothcain  of  equal  moanu  iu  not  lojcctud. 

c.  MI»raAa  Coat  CruwUi  Cotupaiiaon 

The-  Wilooxon  rank  ouin  te.-»t  I  eii  commoii  me-.anu  lor  inlotuKe;  e.-out 
qiowtn  of  Aimy  .uid  Navy  iiyuteimii  -/ieldu  /»  t*jot  utatiuiic  value  of  V'  -  10 
with  a  p  v.aluc?  ol  0.31.  Thin  in  not  uiyni  1  ie.r.'int  and  l  lie  null  liyi)othe3ij  iu 


ai  nmunri  iu  not,  » r?  )eict«?il . 


4.  ^my  S/BtoioM 

a.  Daciaion  Varauo  M.iataka  Coat  Growth 

The  Wilcoxon  rank  oum  tenu  for  comruon  meana  for  decision 
coat  growth  and  inxotake  cont  growth  of  Army  ayote.ino  yields  a  teat 
utatiotic  value  of  W  -  12  v/ith  a  p  value  of  0.045.  Thi.s  is  aignificant 
and  the  null  hyiiothesis  of  equal  meana  is  rejected.  The  DVJA  mistake 
cost  growth  for  7irmy  syaterna  0.004  and  the  DWA  decision  cost  growth  for 

Army  systems  is  0.081. 

b.  Cacagor ical  Aiialyaia 

Table  16  displays  the  dollar  weighted  average  CGI'.a  of  the 
decision  and  itiistake  categories  lor  Army  syotv.n\n .  Uo  statistical  tests 
wore  performed  to  dcteiminc  which  categories  contain  the  largest  cost 
growth;  only  a  visual  inspection  in  requixed. 


TAULH  16.  AVKRAOH  CATB(K)RY  CORO  fOk  ARMY  HYSTKMS 


lie  c  1  s  i  on 

Derv 

bammi 

Oils 

Depf 

Dos  cl 

Dot her 

CGR 

0.079 

0  011 

-0 . 008 

0 

0 

0 

Mrutakfl 

Mcep 

Mcode 

Mils 

Msomt 

Mescl 

Mother' 

CGJ' 

0.004 

-  - 

-0.003 

0 

0.005 

0.002 

•  0 . 004 

The  majority  of  decision  coot  growth  in  Army  systems  occura 
in  this  bciv  category  and  lias  a  dollar  weighted  average  value  of  0,079. 
The  majority  of  mist.rke  coot  growth  in  Army  uyatcina  (.^ccurs  in  two 
cotegorieo,  the  Mce(j  categoj  y  atid  the  Mosmf  category.  Tfie  Mcef)  catc:gory 
has  a  dollar  weigfitcd  average  value,  of  0  004  .  The  Musmf  category  has  a 

dollar  weighted  average  value  of  0.005. 

5 .  Navy  Syatamo 

a.  Dacia  ion  Verana  Mia  taka  Coat  Growth 

Tlie  Wjlcoxon  rank  sum  teat  for  common  muaiifj  for  deci.sion 
cost  growtli  and  miutakc  coat  growtli  of  Navy  systems  yields  a  test 
statistic  vahre  of  W  »  2  3  with  .a  p  value  ol  O.IB.  This  is  not 
slgni'lcant  and  the  null  hypothetils  of  equal  means  is  not  rejected, 
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b.  Categorical  Aamlyaia 

Table  19  displayB  the  dollar  weighted  average  CGRs  of  the 
decision  and  mistake  categories  for  Navy  systems.  No  statistical  test.s 
were  performed  to  determine  which  categories  contain  the  largest  cost 
growth;  only  a  visual  inspection  is  required. 

TABLE  19.  AVERAGE  CATEGORY  CORE  FOR  NAVY  SYSTEM.9 


Decision 

Derv 

Dsmmi 

Dils 

Depf 

r 

Doscl 

CGR 

-0.002 

0.074 

0 . 001 

-0 . 001 

0 . 002 

0 

Mistake 

Mcep 

Mcede 

Mils 

Mssmf 

Mead 

Mother 

CGR 

0.038 

0 

-0.001 

0 

0.003 

0.001 

The  majority  of  decision  cost  growth  in  Navy  systems  occurs 
in  the  Dsmmi  category  and  has  a  dollar  v/eighted  average  value  of  0.074. 
The  majority  of  mistake  cost  grov/th  in  Navy  systems  occurs  in  the  Mcep 
category  and  has  a  dollar  weighted  average  value  of  0.038. 
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VI.  COKCLOSXONS 


A.  KBW  CATEGORIES 

The  new  categorization  syntema  breaks  out  the  cost  growth  into  more 
categories  than  the  SAR  system.  This  system  is  better  at  detecting 
sources  of  cost  growth  in  a  particular  weapon  system  and  enabling 
correction  of  that  source.  It  also  shows  when  mistakes  or  decisions  are 
the  major  effect  behind  the  cost  growth  of  a  system. 

B,  DOLLAR  WEIGHTED  AVERAGES 

Using  dollar  weighted  averages  adjusts  the  findings  to  account  for 
the  greater  effect  more  expensive  weapon  systems  have  on  biidgetary 
considerations.  This  allows  for  a  more  realistic  interpretation  of  the 
cost  growth  and  its  effect  on  the  defense  budget.  If  an  arithmetic 
average  is  used,  the  data  i:i  equally  weighted,  giving  the  cost  growth  of 
low  cost  systems  an  equal  weight  on  the  overall  cost  growth.  This  would 
tend  to  skew  the  results  and  not  allow  them  to  be  applied  directly  to 
the  entire  military  budget. 

An  example  of  this  effect  occurs  when  evaluating  the  decision  and 
mistake  cost  growth  ratios  of  all  the  weapon  systems.  If  an  arithmetic 
average  is  used,  the  average  decision  cost  growth  ratio  is  0.040  and  the 
average  mistake  cost  growth  ratio  is  0.074.  This  is  opposite  to  the 
findings  using  dollar  weighted  averages,  where  the  decision  cost  growth 
ratio  is  0.075  and  the  mistake  co.st  growth  ratio  is  0.033.  3y  using  an 
arithmetic  mean,  it  would  lead  one  to  conclude  that  7.4%  mistake  cost 
growth  is  occurring  in  the  defense  budget,  when  in  reality,  only  3.3%  is 
occurring.  Therefore,  dollar  weighted  averages  provide  a  more  realistic 
picture  of  cost  growth. 
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C.  SYSTEM  COST  GROWTH 


1.  Total  Cost  Growth 

The  total  cost  growth  of  a  weapon  system  is  much  less  than  might 
be  expected.  It  has  a  dollar  weighted  average  value  of  C.108. 

A  1980  study  reports  that  the  average  total  cost  growth  of 
we-ipons  systems  is  25. 9%  [Ref,  3],  This  is  significantly  higher  than 
the  10.8%  this  analysis  shows.  However,  the  time  of  that  study  is 
relevant  to  the  difference  in  findings.  The  development  by  estimators 
of  a  computerized  large  data  base,  which  contains  the  production  costs 
of  a  large  number  of  components,  assemblies,  and  subassembli.es,  allows 
for  a  more  accurate  estimate  of  the  production  costs  of  new  systems. 

This  is  beneficial  in  estimating  system  costs  which  should  bring  cost 
growth  down.  Another  cause  is  the  high  technologies  that  v;ere  brand  new 
in  1980  and  extremely  difficult  to  estimate  have  been  developed  and 
produced  so  that  now  they  are  much  easier  to  estimate.  Lastly,  the 
results  of  the  1980  study  and  others  re.sulted  in  a  determined  effort  by 
the  Department  of  Defense  to  control  cost  growth.  This  has  also  had  a 

positive  impact  in  lowering  the  cost  growth  of  weapons  systems, 

2.  Syetain  Model 

Eighty  five  percent  of  the  total  cost  growth  can  be  captured  in 
Dsmmi  and  the  Mcep  categories.  To  control  total  cost  growth,  it  is 
required  that  the  cost  growth  in  theoe  two  categories  be  focused  upon. 
The  possible  causes  of  the  cost  growth  in  these  categories  will  be 
discussed  below. 

3.  Mistake  and  Doclalon  Coat  Growth  Comparison 

Mistakes  make  up  30,6%  of  the  total  cost  growth  of  a  system 
while  decisions  make  up  69.4%  of  the  total  cost  growth.  Therefore 
decisions  outweigh  mistakes  by  a  margin  of  2.3:1. 

One  recent  study  reports  that  cost  growth  due  to  deci.sions 
outweighs  cost  growth  due  to  mistakes  by  a  margin  of  two  to  one  [Ref. 

7] .  This  analysis  reaches  a  similar  conclusion. 
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4.  Mldtak*  Cost  Growth 


A  majority  of  the  mistake  cost  growth  occurs  in  the  Mcep 
category.  The  Mcep  category  is  used  to  classify  cost  variances 
attributable  to  mistakes  in  estimating  the  cost  of  production.  While  it 
may  be  noted  that  ,his  implies  that  the  Department  of  Defense  needs  to 
concentrate  its  efforts  on  becoming  better  at  estimating  the  production 
costs  of  a  wear  system,  the  mistake  cost  growth  for  a  system  is  only 
3.3%  and  is  nc  nearly  as  significant  as  controlling  the  cost  growth  due 

to  decisions , 

5.  Decit  ‘.on  Cost  Growth 

A  ma.ority  of  the  decision  cost  growth  occurs  in  the  Dsmmi 
category.  The  Dsmmi  category  is  used  to  classify  cost  variances 
attributable  to  a  decision  to  change  the  procurement  schedule,  shifts  in 
the  multiyear  procurement  rate  or  in  different  management  initiatives. 

A  detailed  analysis  of  the  data  indicates  that  a  majority  of  the  Dsmmi 
cost  growth  is  due  to  schedule  slippage.  Some  of  this  schedule  slippage 
can  be  attributed  to  decisions  to  change  the  design  and  performance 
requirements  of  the  system.  These  changes  are  classified  into  the  Dcrv 
category  and  has  a  dollar  weighted  value  of  0.015.  This  may  or  may  not 
account  for  all  of  the  schedule  slippage,  however,  the  data  does  not 
provide  sufficient  evidence  to  support  or  deny  thi.s  possible 
explanation.  Further  study  concentrating  on  the  causes  of  schedule 
slippage  needs  to  be  done. 

D.  SYSTEM  COST  COMPARISON 

The  driving  factor  between  the  difference  between  low  cost  system 
cost  growth  and  high  cost  system  cost  growth  is  the  difference  in  the 
mistake  cost  growth. 

The  low  cost  systems  have  a  higher  mistake  cost  growth  by  a  margin 
of  2.4:1.  Beth  high  and  low  cost  systems  have  a  majority  of  their 
mistake  cost  growth  occurring  in  the  Mcep  category.  This  is  not 
surprising  since  this  result  has  already  been  determined  above. 
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However,  low  cost  systems  have  over  twice  as  much  cost  growth  in  the 
Mcep  category  than  do  high  cost  systems.  This  indicates  that  estimators 
are  not  nearly  as  adept  at  estimating  the  production  costs  of  low  cost 
systems  as  high  cost  systems. 

One  possible  explanation  for  this  is  that  estimators  do  not  take  as 
much  care  in  estimating  the  costs  of  low  cost  systems.  Low  cost  systems 
are  not  nearly  as  visible  on  a  Congressional  oversight  level  or  cn  an 
internal  level,  and  therefore  the  care  taken  to  accurately  estimate  the 
production  cost  of  a  system  may  not  be  nearly  as  high  as  for  that  of  the 
more  expensive  systems. 

Another  explanation  concerns  the  nature  of  high  cost  to  low  cost 
systems  [Ref.  1] .  A  majority  of  high  cost  systems  are  large  platforms 
which  have  a  majority  of  their  production  cost  resulting  from  large 
components.  These  large  components,  such  as  hull  construction  or 
propulsion  plants,  have  been  used  before  or  a  similar  component  has  been 
used  before.  This  makes  the  estimation  of  the  production  costs  of  these 
large  components  fairly  accurate.  The  mistakes  in  estimating  the 
smaller  and  relatively  cheaper  items  would  be  masked  in  the  overall  cost 
growth  of  the  system  even  if  they  were  of  equal  dollar  value  to  the 
errors  in  the  low  cost  systems . 

Therefore,  the  difference  between  the  low  cost  and  high  cost  system 
mistake  cost  growth  is  probably  not  as  significant  as  it  appears  to  be. 

B.  MISSILB  SYSTEMS 

The  significant  difference  between  the  cost  growth  of  missile 
systems  as  compared  to  other  weapon  systems  is  that  the  mistake  cost 
growth  for  missile  systems  is  significantly  less  than  that  of  other 
systems,  specifically  in  estimating  the  cost  of  production.  The  most 
likely  explanation  is  similar  to  the  discussion  above  concerning  the 
cost  of  a  system.  Missiles  are  evolutionary  in  nature.  The  guidance 
system  of  newer  missiles  is  generally  a  modernization  or  modification  of 
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a  previous  guidance  system.  The  same  can  be  said  for  propulsion  system 
and  '.varhead  type.  This  would  tend  to  increase  the  accuracy  of  missile 
systems  over  other  types  of  systems. 

One  report  on  cost  growth  states  that  missile  systems  have  a  much 
higher  cost  growth  than  do  other  systems  [Ref.  l] .  This  article, 
prepared  in  1982,  included  much  older  missile  systems.  The  evolutionary 
process  discussed  above  had  not  yet  developed  and  therefore  a  higher 
cost  growth  could  have  resulted.  A  comparison  of  the  cost  growth  of 
older  missile  systems  to  newer  ones  would  be  required  to  validate  this 
assertion. 


F.  SERVICE  COMPARISON 


The  Army  and  Navy  have  effectively  equivalent  cost  growths.  The 
only  difference  is  that  part  of  the  Army's  mistake  cost  growth  results 
from  mistakes  in  the  estimation  process  that  resulted  in  schedule  slips. 


future . 

A  majority  of  early  1980s  studies  reported  that  the  Army  had  a 
significantly  higher  cost  growth  than  the  Navy  [Refs.  1,  3] .  This  was 
attributed  to  the  uiodernization  of  the  Army.  This  study  concentrated  on 
more  recent  weapons  systems  and  did  not  find  this  to  be  the  case.  The 
most  obvious  explanation  for  this  effect  is  that  the  Army  has  completed 
its  modernization  and  is  now  proficient  at  controlling  the  costs  of  its 
weapon  systems . 
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VII.  RbCOMMSKDATIONS 


A.  NEW  CATEGORIES 

The  new  categorization  system  should  be  used  to  cla.ssify  cost 
variances  in  the  SAR  system.  This  would  allow  for  a  more  detailed 
accounting  of  the  cost  variances  and  allow  for  identification  of  tne 
exact  areas  in  which  cost  growth  is  occurring. 


B.  TOTAL  COST  GROWTH 

The  total  cost  growth  as  been  significantly  lowered  over  the  past 
ten  years.  This  is  significant  and  promising.  The  Department  of 
Defense  should  continue  its  current  oversight  an  decision  process  to 
ensure  that  future  cost  growth  is  maintained  at  acceptable  levels. 


C.  DECISION  COST  GROWTH 


Decision  cost  growth  is  the  driving  force  behind  the  total  cost 


growth  of  weapon  systems.  This  is  driven  by  decisions  that  result  in 
schedule  slippage  in  the  procurement  process.  This  tj/pe  of  cost  growth 
must  be  minimized.  The  most  likely  solution  is  to  ensure  at  Milestone 
II  that  all  decisions  that  can  be  made  that  could  result  in  schedule 
slippage  are  made  and  finalised. 


D.  MISTAKE  COST  GROWTH 

The  mistake  cost  growth  is  well  within  any  reasonable  limits .  The 
driving  factor  for  this  type  of  growth  is  mistakes  in  estimating  the 
production  costs  of  a  weapon  system.  This  amounts  to  only  3.3t  co.st 
grov;th  ana  is  well  within  any  reasonable  margin  of  error  that  could  be 
placed  on  this  type  of  estimation. 


E.  AKiiy  COST  GROl^TH 

The  Army  mistake  cost  growth,  while  similar  in  magnituda  to  the 
Navy's,  is  partially  driven  by  schedule  slippage.  The  Army  must 
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evaluate  the  factors  that  are  causing  this  problem  and  correct  them. 

One  possible  solution  is  to  compare  the  Army's  estimation  process  with 
the  Navy's  and  to  attempt  to  isolate  the  key  difference.s  that  result  in 
this  type  of  cost  growth. 

F .  FDTURE  STUDIES 

All  of  the  results  and  conclusions  of  this  study  are  tentative  since 
they  are  based  on  nine  weapon  systems.  This  is  the  initial  analysis  of 
a  three  year  study  by  the  DoD  and  the  results  and  conclusions  of  the 
study  using  all  of  the  SAR  reportable  weapon  systems  may  differ  from 
this  report. 

A  continuation  of  this  study,  once  data  for  more  weapons  systems  has 
been  obtained,  should  be  done  to  compare  Air  Force  systems  with  those  of 
the  Army  and  Navy.  Also,  a  comparison  of  aircraft  and  ship  cost  growth 
would  be  beneficial. 

A  study  of  RDT&E  cost  growth  is  necessary  to  evaluate  the  decision 
and  estimation  proce.93  of  this  aspect  of  weapon  systeu  acquis itiv*n . 
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